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Abstract 

We express the averages of products of characteristic polynomials for random matrix ensembles as- 
sociated with compact symmetric spaces in terms of Jack polynomials or Heckman and Opdam's Jacobi 
polynomials depending on the root system of the space. We also give explicit expressions for the asymp- 
totic behavior of these averages in the limit as the matrix size goes to infinity. 
MSC-class: primary 15A52; secondary 33C52, 05E05. 

Keywords: characteristic polynomial, random matrix, Jacobi polynomial, Jack polynomial, Macdonald 
polynomial, compact symmetric space. 

1 Introduction 

In recent years, there has been considerable interest in the averages of the characteristic polynomials of 
random matrices. This work is motivated by the connection with Ricmann zeta functions and L-functions 
identified by Keating and Snaith [KS1, KS2]. The averages of the characteristic polynomials in the cases of 
compact classical groups and Hermitian matrix ensembles have already calculated, see [Me] and references in 
[BG]. In these studies, Bump and Gamburd [BG] obtain simple proofs for the cases corresponding to compact 
classical groups by using symmetric polynomial theory. Our aim in this note is to use their technique to 
calculate averages of the characteristic polynomials for random matrix ensembles associated with compact 
symmetric spaces. 

We deal with the compact symmetric spaces G/K classified by Cartan, where G is a compact subgroup 
in GL(N, C) for some positive integer N, and K is a closed subgroup of G. Assume G/K is realized as a 
subspace S in G, i.e., S ~ G/K, and the probability measure dM on S is then induced from G/K. We call 
the probability space (S, dM) the random matrix ensemble associated with G/K. 

For example, U(n)/0(n) is the symmetric space with a restricted root system of type A, and is realized 
by S — {M G U(n) | M = M T }. Here M T stands for the transposed matrix of M while U(n) and 0(n) 
denote the unitary and orthogonal group of matrices or order n respectively. The induced measure dM on 
S satisfies the invariance d(HMH ) = dM for any H G U(n). This random matrix ensemble (S, dM) is 
well known as the circular orthogonal ensemble (COE for short), see e.g. [Dy, Me]. 

We also consider the classical compact Lie groups U(n), SO(n), and Sp(2n). Regarding these groups as 
symmetric spaces, the random matrix space S is just the group itself with its Haar measure. 

The compact symmetric spaces studied by Cartan are divided into A and BC type main branches ac- 
cording to their root systems. There are three symmetric spaces of type A, with their corresponding matrix 
ensembles called circular orthogonal, unitary, and symplectic ensembles. For these ensembles, the probability 
density functions (p.d.f.) for the eigenvalues are proportional to 

A Jack (z;2//3) = [] \zi-zjf, 

l<i<j<n 
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with (5 = 1, 2, 4, where z = (zi, . . . , z„), with = 1, denotes the sequence of eigenvalues of the random 
matrix. We will express the average of the product of characteristic polynomials det(7 + xM) for a random 
matrix M as a Jack polynomial ([Mac, Chapter VI-10]) of a rectangular-shaped Young diagram. Jack 
polynomials are orthogonal with respect to the weight function A Jack . Our theorems are obtained in a 
simple algebraic way, and contain results given in [KS1]. 

For compact symmetric spaces of type BC root systems, the corresponding p.d.f. is given by 

A H °(z;fc 1 ,A ;2 ,fc3)= n ii-^7 i i 2fc3 ii-^i 2fc3 - n ii-^nwr 2 - 

l<i<j<n l<j<n 

Here the fc,'s denote multiplicities of roots in the root systems of the symmetric spaces. For example, 
the p.d.f. induced from the symmetric space SO(4n + 2)/{SO(An + 2) n Sp(4n + 2)) is proportional to 
A HO (z; 2, i, 2). For this class of compact symmetric spaces, Opdam and Heckman's Jacobi polynomials 
([Di, HS]), which are orthogonal with respect to A HO , will play the same role as Jack polynomials for type 
A cases. Namely, we will express the average of the product of characteristic polynomials det(J + xM) as 
the Jacobi polynomial of a rectangular-shaped diagram. 
This paper is organized as follows: 

Our main results, which are expressions for the averages of products of characteristic polynomials, will 
be given §6. As described above, the symmetric spaces corresponding to the two root systems, type A and 
BC, will be discussed separately. For type A spaces, we use Jack polynomial theory. These discussions can 
be generalized to Macdonald polynomials. Thus, after preparations in §2, we give some generalized iden- 
tities involving Macdonald polynomials and a generalization of the weight function A Jack in §3 and §4. In 
particular, we obtain g-analogues of Keating and Snaith's formulas [KS1] for the moments of characteristic 
polynomials and a generalization of the strong Szego limit theorem for Toeplitz determinants. These identi- 
ties are reduced to characteristic polynomial expressions for symmetric spaces of the A type root system in 
§6.1 - §6.3. On the other hand, for type BC spaces, we employ Opdam and Heckman's Jacobi polynomials. 
We review the definition and several properties of these polynomials in §5, while in §6.4 - §6.12 we apply them 
to obtain expressions for the products of characteristic polynomials of random matrix ensembles associated 
with symmetric spaces of type BC. 



2 Basic Properties of Macdonald symmetric functions 

We recall the definition of Macdonald symmetric functions, see [Mac, Chapter VI] for details. Let A be a 
partition, i.e., A = (Ai, A 2 , . . . ) is a weakly decreasing ordered sequence of non-negative integers with finitely 
many non-zero entries. Denote by ^(A) the number of non-zero Xj and by |A| the sum of all Xj. These values 
£(X) and |A| are called the length and weight of A respectively. We identify A with the associated Young 
diagram {(i, j) e Z 2 | 1 < j < Xi}. The conjugate partition A' = (A' 1; A 2 , . . . ) is determined by the transpose 
of the Young diagram A. It is sometimes convenient to write this partition in the form A = (l™ 1 ^™ 12 • • • ), 
where mi = to^A) is the multiplicity of i in A and is given by rrii = X ■ — A- +1 . For two partitions A and fi, 
we write A C [i if A^ < ^ for all i. In particular, the notation A C (to™) means that A satisfies Ai < m and 
Ai < n. The dominance ordering associated with the root system of type A is defined as follows: for two 
partitions A = (Ai, A 2 , . . . ) and n. = (/ii, ,u 2 , • • ■ ), 

M <a A <=> |A| = and (J,i + • • • + m < \i + • • • + \i for all i > 1. 

Let q and t be real numbers such that both \q\ < 1 and |i| < 1. Put F = Q(q,t) and T™ = {z = 
(z\, . . . , z n ) | \zi\ = 1 (1 < i < n)}. Denote by F[x\, . . . , x n ] 6n the algebra of symmetric polynomials in 
variables x\, . . . , x n . Define an inner product on F[x\, . . . , x n ] 6n by 

(/,5>AMao = i f /( z ) 5 (z- 1 )A Mac (z;g,t)dz 



with 



A Mac (z;g,t)= [] 

l<i<j<n 



(tZiZj 1 :q) c 
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where z 1 = (z 1 , . . . , z n 1 ) and (a; q)^ = n^o(l — a( f )- Here dz is the normalized Haar measure on T". 
For a partition A of length £(A) < n, put 

(2.1) m$(x lt ...,x n )= J2 x i 

L/=(yi,...,y„)es n A 

where the sum runs over the 6„-orbit 6„A = {(A CT (i), . . . , A CT („)) | cr e 6„}. Here we add the suf- 
fix "A" because & n is the Weyl group of type A. Then Macdonald polynomials (of type A) i^ ac = 
P x ac (xi, . . . , x n ; q, t) <G F[xi, . . . , x n ] &n are characterized by the following conditions: 

P A Mac = m£ + with u ^ e F ' ( P A MaC , < ac >AMao =0 if A ? ii. 

Denote by A^ the P-algebra of symmetric functions in infinitely many variables x — (x\, x 2 , . . . ). That 
is, an element / = f(x) S Ap is determined by the sequence (f n )n>o of polynomials /„ in F[xi, . . . , x„] e ", 
where these polynomials satisfy sup Il>0 deg(/„) < oo and f m (xi, . . . , x n , 0, . . . , 0) = f n (xi, ■ ■ ■ , x n ) for any 
m > n, see [Mac, Chapter 1-2]. Macdonald polynomials satisfy the stability property 



Pf ac (x 1: ...,x n ,x n+1 ;q,t) = Pf ac (x l7 ...,x n ;q,t) 

x n+1 =0 

for any partition A of length l(X) < n, and therefore for all partitions A, Macdonald functions P x Max: (x; q, t) 
can be defined. 

For each square s — of the diagram A, let 

a(s) = \i-j, a'(s)=j-l, l(s) = X' j -i, l'(s)=i-l. 

These numbers are called the arm-length, arm-colength, leg-length, and leg-colength respectively. Put 

cx(q,t) = H(l - q< s H l ^ +1 ), c' x (q,t) = [](1 - q a(s)+1 t 1 ^). 

sGA sGA 

Note that c\(q,t) = c' x ,(t,q). Defining the Q-function by Q\(x;q,t) — c\(q,t)c' x (q,t)~ 1 P\(x;q,t), we have 
the dual Cauchy identity [Mac, Chapter VI (5.4)] 

(2.2) ^P A (a;; g ,0Pv(y;i>9) = ^QA(^g,t)Qv(y;*>5) 

A A 

= n n^ 1 + Xiv ^ = ° xp ( yi — i — Pk{x)pk{y) j , 

i>lj">l \fe=l / 

where y — (j/i, j/2, • • ■ )• Here pk is the power-sum function pk{x) = x\ + x% H . 

We define the generalized factorial (a) x q ^ by 

(# } = H(t l 'W -q a '^a). 

sex 

Let u be an indeterminate and define the homomorphism e Ujt from Ap to F by 

1 - u r 

(2.3) euAPr) = j— p for a11 r ^ L 

In particular, we have £*«*(/) = /"(l, t, t 2 , - - - , t" - x ) for any / e Ai?. Then we have ([Mac, Chapter VI 
(6.17)]) 

(2.4) e ., ( pr-) = g^. 

Finally, the following orthogonality property is satisfied for any two partitions A and [i of length < n: 

(2-5) (Pf^, Q^ aC ) A Mao = 5 Xfl (l, l) A Mac [] - _ + 1 ,,(,,_! ■ 

s£A ^ 
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3 Averages with respect to A Mac (z; q,t) 

As in the previous section, we assume q and t are real numbers in the interval (—1,1). For a Laurent 
polynomial / in variables z\, . . . , z n , we define 

[J>n J Tn A^(z;q,t)dz ' 

In this section, we calculate averages of the products of the polynomial 

n 

* A (z;77) = TJ(l + ^), ry £C 

with respect to {-)n^ ■ Denoting the eigenvalues of a unitary matrix M by zi,...,z n , the polynomial 
\E ,A (z; rf) is the characteristic polynomial det(7 + r}M). 

The following theorems will induce averages of the products of characteristic polynomials for random 
matrix ensembles associated with type A root systems, see §4 and §6.1 - §6.3 below. 

Theorem 3.1. Let K and L be positive integers. Let r/i, . . . , t)l+k be complex numbers such that r/j ^ 
(1 < j < L). Then we have 

I L K \ («'*) 

( [] * A (z -1 ; vi 1 ) • n ) = fa ■ ■ ■ ^)" n • fa> ■ ■ ■ , vl+k; t, q ). 

\l=l k=l I n 

Proof. By the dual Cauchy identity (2.2), we have 

n ^(z-^vr 1 ) • n * a (^ vL +k ) = n ir ■ fa • • • ■ L ff na + ^ 

1=1 fe=l 1 = 1 fe=l j=l 

L 



= LI < n ■ fa ■ ■ ■ z ^ L E ^fa' ■ ■ ■ , Vl+k; t, q)Qf, ac (z; q, t). 



i=i 



Therefore, since Pffi(z;q,t) = (zi ■ ■ ■ z n ) L ([Mac, Chapter VI (4.17)]), we see that 

mvH*- 1 ;* 1 ) • n * A fa v L+k ) ) = n *r E ^ Iac fa> ■ ■ ■ . vl+k *. g) - ^''r ac 

\r=l fe=l /„ S=l A (1,1) A Ma. 

=n»r-^)('h.-.^*'«) n i_ J. )+ i t ,-i> M -i 

z=i se(L") y 

by the orthogonality property (2.5). It is easy to check that 

1 - q a'(s) t n-l'(s) _ c {Ln) (q,t) _ c[ nL) {t,q) 
} { l n) l-q a '^ +1 t n -ns)-i J {Ln) (q, t ) C (nL) {t,qY 

and so we obtain the claim. □ 

It may be noted that the present proof of Theorem 3.1 is similar to the corresponding one in [BG]. 
Corollary 3.2. For each positive integer k and £ <G T, we have 

l k - 1 \(i,t) k_i n _i k+i+l +j 

(ni* A (^ +i/2 oi 2 ) -UU-r^rf 

\i=0 l n i=0 j=0 H 
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Proof. Set L = K = k and rji 1 = rj l+k = q % 1 / 2 £ (1 < i < k) in Theorem 3.1. Then we have 

\ («.*) fc-1 



/fe-1 



H |* A (*; g*+V2 |2 \ = J] g (i + l/2)n . (g - fc+ l/ 2j g - fc+ 3/2 ) _ ) g -l/2 } g l/ 2) . . . ^-1/2.^ g) 



i=0 



_ /1 nfe 2 /2 „(-fc+l/2)fenp /-i _ _2k-l. + ,,\ 



-nfe(fe-l)/2 



l 2k ,q(P(n k )(-',t, q))- 



From expression (2.4), the right-hand side of the above expression equals 



-nfe(fe-l)/2 



( fl 2k\(t,?) 

W 7 (» fc ) _ -nfe(fe-l)/2 



1111 1 _ fn-jqk-i+1 



nn^ 



C( „ fc) (t, g) " 11 11 1 - tn-i g fc-i+l ±± 11 1 - tJg*-<+l ' 

and the result follows. □ 
Kaneko [K2] defines the multivariable g-hypergeometric function associated with Macdonald polynomials 



by 



2 $i (9 ' t) (a,6;c;xi,...,a; Il ) = ^ 



(<(,*) 



where A runs over all partitions of length £(X) < n. The g-shifted moment (J{ k = ^ \%> A {z; q l+1 / 2 £)\ 2 ' 

given in Corollary 3.2 can also be expressed as a special value of the generalized g-hypergeometric function 
2$i ( '''* ) as follows: 

Proposition 3.3. For any complex number with \r}\ < 1 and real number u, 

2\ (9.*) 



\J3 



(rjzju; q) c 



} 



a*!^*)^- 1 ^- 1 ;^"- 1 ; (u\rj\) 2 , (u\n\) 2 t, M^V" 1 ). 



In particular, letting u — q k and r] = q 1 ^ 2 ^ with £ e T, we have 



H \* A (z: q l+1 ' 2 0? ) = 2<*>l (q < t) (q-\ q- k ; fl*"" 1 ; q 2k+ \ q 2k+ H, q^+H^ 1 ). 



\i=0 



Proof. A simple calculation gives 

{vZj;q)o, 



n 



1 = 1 {■ n z j u;q) oc 
From expressions (2.2) and (2.3), we have 

(vzj;q)oo 



I 00 (_-nfc-l 1 _ u k N 



n 



\ (vzju;q) c 



Thus we have 



n 



(■qzju; q) c 



= E(-^) |A|e ^Wv ac (-; t, q))Qf ac (z ; 9 , t) = E(-^) |A ' e ^(^ Iac (-; t, 9))^A Mac (^; 3, t). 



(-v) M e u , q (P^(-,t, q))e us {Q^{-- 1, q))P^ c {z; q, ^Q^z' 1 ; q, t). 
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The average is given by 



n 



(nzju; q) c 



(<?,*) 



= E \v\ 2lXl e u , q (Py ac (-, *, q))e u , q (Qy ac (-: *, q)) 
x 



I pMac nMac\ 
Mac/ . . „\\\f\ L^jj 



(1, l)^Mac 



-Vl n .2iAi {(")£' g) > 2 n 

^ 1 " cy(t,q)c' x ,(t,q) 11 1 - g«'M+li»-J'M-l 

by expression (2.4) and the orthogonality property (2.5). It is easy to check that 

(u)% q) = (-u^iu- 1 )^, c x ,(t,q)c' y (t,q) = c x (q,t)c' x (q,t), 



sex 



a'(s) + l£n — V (s) 



v(s)-i n fv 



seX 



ll'(s) _ go,' (s) + ljn-l (qt n ^ 1 ) x q '^ ' 



Finally, we obtain 



(vzj-,q)c 



{rjZjU; q) c 



(9,*) 



= E(«H) 2|A| 



{(«- 1 )i a '* ) } 2 ^ BC (i,*.---,* n - 1 ;9,t) 



(^- 1 ) A 9 



which equals 2 $i (9 '* ) (u~ 1 > qt 71 ' 1 ; {u\n\) 2 , (u^l) 2 *"" 1 ). 



□ 



Now we derive the asymptotic behavior of the moment of \^f(z; n)\ when < 1 in the limit as n — > oo. 
The following theorem is a generalization of the well-known strong Szego limit theorem as stated in §4.2 
below. 

Theorem 3.4. Let <f>(z) = exp(J] fc6Z c(k)z k ) be a function on T and assume 

(3.i) Ei c ( fc )i <o ° and Ei fc H c W 2 < °°- 

feez fcez 

Then we have 

' («.*) / oo , 

1 - g ft 



lim e-" c (°) ( T\<t>{zj) 

\j=i / 



(oo ^ 
5^fcc(fc)c(-fc) — 
fe=l 



t k 



Proof. First we see that 



\{4>{z j )=e n ^\{ C Mc{k)vk{z)) C Mc{-k)p k (z)) 

i=i fe=i 



fc=l \a=0 



\b=0 



- nc(0) E E (fl ^^ )p(i-^--oW^^. 

(l»12 a 2... ) (1&12&2... ) \fe=l / 

where both (l ai 2 a2 • • • ) and (l fcl 2 fc2 • • • ) run over all partitions. Therefore we have 

e- nc ^lf[4>{z j )) q ' t) = E E f ft C(fc)Qfc ^ lai2a2 -)' p ( lbl2 " 2 -)^ AMa 

\j = l /„ (1«12»2...) (l6l 2 i>2...) \fe=l afe ' ^ / 



(1, l) A Mac 
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We recall the asymptotic behavior 



■■■),P(1>i 2 >2-))am.c n / l - g fc \ 

(l,l) A Mac M fc Al-*V 

in the limit as n — > oo, see [Mac, Chapter VI (9.9) and (1.5)]. It follows from this that 



\j=l / „ (l»i2«2...)fe=l 



k=l \a=0 ' I \k=\ 



Here X^fcLi kc(k)c(—k) \_\ k converges absolutely by the second assumption in (3.1) and the Cauchy-Schwarz 

±2 

-I' 



inequality, because | \_ q t k | < i_mfc — + M- ^ 



Note that the present proof is similar to the corresponding one in [BD] . The result in [BD] is the special 
case of Theorem 3.4 with q — t. As an example of this theorem, the asymptotic behavior of the moment of 
|\I> A (z; i]) | is given as follows. A further asymptotic result is given by Corollary 4.2 below. 

Example 3.1. Let 7 <G R and let rj be a complex number such that \n\ < 1. Then we have 

M,t) _ ( {q\v\ 2 ,t)c 



hm (\* A {»;r,)n™ = ( { - 



J OO 



M 2 ;i)c 



This result is obtained by applying Theorem 3.4 to <j)(z) = |1 + ?7z| 27 . Then the Fourier coefficients of log0 
are c(k) = (-l) fc -V7A and c(-fc) = (— l) fc - 1 r7 A: ^/fc for fc > 0, and c(0) = 0. □ 

4 Circular ensembles and its g-analogue 

4.1 Special case: t = g^ 2 

In this subsection, we examine the results of the last section for the special case t — q^l 2 with > 0, i.e., 
we consider the weight function A Mac (z; q, <j' 3 / 2 ). Denote by (■)' ^ the corresponding average. Define the 
g-gamma function (see e.g. [AAR, (10.3.3)]) by 

T q (x) = (l-q) 1 -* 



(q x ;q)oo 

Proposition 4.1. Let (3 be a positive real number. For a positive integer k and £ € T, we have 

( } VU 11,9 ii r t( |(* + 4 + i))r t( „ + |(i + 1)) (mtht q } 



gr,(f J + 2Hi)r,(f J + i) 
3=0 



Proof. The claim follows immediately from Corollary 3.2 and the functional equation T q (l + x) = r,(x). 

□ 

Consider now the asymptotic behavior of this average in the limit as n — > 00. Put [n] q = (1 — (?")/(! — q). 
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Corollary 4.2. For a positive integer k and £ eT, it holds that 

(4.2) limanlO-^/ni^^OI'V = ff FT^T^ * = ^ 

\r=i /„ ;/3 « 1 i r *(f( fc + * + 1 )) 

Proof. Verify that 

V ' r t (n)([n] t )« 

for any constant a. Then the claim is clear from expression (4.1). □ 
Example 4.1. Denote by .Fg(fc) the right-hand side of equation (4.2). Then we obtain 

fc-i [2j + l] i! 

(4.4) = n ■ 97 



(-) ^=n^ 



(4-6) ^(2fc)= T g^ T 2 ff 



[2fc-ip ^ [2j] q V 

Here [n] 9 ! = [n] q [n - l] q ■ ■ ■ [l] q and [2k - l] q l\ = [2k - l] q [2k - 3] 9 ■ ■ ■ [3] q [l] q . Equalities (4.4) and (4.5) are 
trivial because T q (n + 1) = [n] q \. We check relation (4.6). By definition, we have 

T H2k\ = T7 EdMi±12j = rf iy(p + f)i>(p + i) 

to Mfc + §(i + i)) llr, a (fc + p + i)r, a (A ! +p + i)- 

Using the g-analogue of the Legendre duplication formula (see e.g. [AAR, Theorem 10.3.5(a)]) 

T q (2x)T q2 (1/2) = (1 + q) 2x - x Y q , (x)T q2 (x + 1/2), 

we have 

4( )_ M r 9 (2fc + 2 P + i) - ([2W llpfc+^j- 

Expression (4.6) can then be proven by induction on n. 

4.2 Circular /3-ensembles and Jack polynomials 

We take the limit as q — > 1 of the results of the previous subsection. Recall the formula 

lim^^ = (l-x)- a 

for | as | < 1 and a € R, see [AAR, Theorem 10.2.4] for example. Then we have 

lim A Mac (z;q,qM 2 ) = TT k - z, =: A Jack (2; 2//3), 

9— »1 -*--*- 

l<2<j<n 

which is a constant times the p.d.f. for Dyson's circular /3-ensembles (see §6). Denote by (-) n ,/3 the corre- 
sponding average, i.e., for a function / on T" define 
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Let a > 0. The Jack polynomial (xi, . . . ,x n ;a) for each partition A is defined by the limit ap- 

proached by the corresponding Macdonald polynomial, 

P A Jack (zi, ...,*„; a) = lim P^fa, ...,x n ;q, q 1/a ), 

q^l 

see [Mac, Chapter VI- 10] for detail. Jack polynomials are orthogonal polynomials with respect to the weight 
function A Jack (,z; a). In particular, s\(x\, . . . , x n ) — P^ ack (x\, . . . , i„; 1) are called Schur polynomials, and 
are irreducible characters of U(n) associated with A. 

From the theorems in the last section, we have the following: from Theorem 3.1, we see that 

(4.7) m * A (^" 1 ; vr 1 ) ■ II * A ( z ^ VL+k)) = (m ■ ■ ■ v L )- n ■ Pffiivu vl+k; P/2). 
\i=i k=i / n>/3 

For a positive real number 7 and complex number i] with |?7| < 1, we have from Proposition 3.3 that 

(4-8) <|tt(z; v)n n , C0E = 2 Fi (2//?) (-7, -7; §(n - 1) + 1; M 2 , ■ • • , M 2 ), 

where (a, b; c; x\, . . . , x n ) is the hypergeometric function associated with Jack polynomials [Kl] defined 

by 

a Ha c ^ aj 

with 

MA a) = I[( u - r ( s )/ a + a '( s ))' and c' A (a) = n(«W«) + 1 )+^))- 

s£A s£A 

For a positive integer A;, and £ € T, by Theorem 4.1 and Corollary 4.2 it holds that 

/ A ,tv " T T lr (!( J + 1 )) r ("+K fc + *+!)) W r(§(* + l)) „, 2/ „ 

(«) <i.*(«or>^ - n r( | (t+i + 1))r( : +f(l + 1)) ~ n fxiffrr+iiy ' /s 

in the limit as n — ► 00. For a function </>(z) = exp(^ fceZ c{k)z k ) on T satisfying inequalities (3.1), by 
Theorem 3.4 it holds that 

(4.10) lim e~ n <V (j\ct>( Zj )J = exp fcc(fc)c(-fc)) ■ 

In particular, for 7 G R and a complex number 77 such that < 1, we have 

lim (|* A (z;^)| 2 T) n =(1-H 2 )"^. 

Several observations may be made concerning the above identities: equation (4.8) is obtained by verifying 
the limits 

T (9 a )i 9,t) |A|r ,(a) ,. c 'aM) / / > 



with q — t a . The expression for the moment is obtained in [FK] using a different proof, which employs a 
Selberg type integral evaluation. Equation (4.9) is also obtained in [KS1] essentially by the Selberg integral 
evaluation. When (3 = 2, equation (4.10) presents the strong Szego limit theorem for a Toeplitz determinant. 
Indeed, the average of the left-hand side of (4.10) is then equal to the Toeplitz determinant det(di-j)i<i,j<n 
of (j), where di are Fourier coefficients of <p. Equation (4.10) with general (3 > is seen in [Jl, J2], but it may 
be noted that the present proof, employing symmetric function theory, is straightforward. This expression 
is applied in [Mat] in order to observe an asymptotic behavior for Toeplitz 'hyperdeterminants'. 
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5 Jacobi polynomials due to Heckman and Opdam 

The results obtained in §3 and §4 will be applied to random matrix polynomials from symmetric spaces 
of the type A root system in the next section. In order to evaluate the corresponding polynomials of the 
BC type root system, we here recall Heckman and Opdam's Jacobi polynomials and give some identities 
corresponding to (4.7) and (4.9). 

The dominance ordering associated with the root system of type BC is defined as follows: for two 
partitions A = (Ai, A2, . . . ) and /1 = (1^1,^2, ■ ■ ■), 

fi < A hi H + /ij < Ai H + Aj for all i > 1. 

Let C[a; ±1 ] = Clxf 1 , . . . , x^ 1 ] be the ring of all Laurent polynomials in n variables x — (xi, . . . , x n ). The 
Weyl group W — 7L 2 I & n = Z 2 x> & n of type BC n acts naturally on Z™ and Cfa^ 1 ], respectively. Denote by 
Cfa^ 1 ]^ the subring of all VF-invariants in C[a; ±1 ]. Let A HO (;z; k\, k 2 , k 3 ) be a function on T" defined by 



z}\^ 



A*°( Z ;k u k 2 ,k 3 ) = J] ^-ZiZj^l-ZiZj^- n ii-^i 2fci ii 

\<i<j<n l<j<n 

Here we assume fci, k 2 , and k 3 are real numbers such that 

h + k 2 > -1/2, k 2 > -1/2, k 3 > 0. 
Define an inner product on C[x ±1 ] w/ by 

(/,5)aho = -^ f f(z)g(z- 1 )A }io (z;k u k 2 ,k 3 )dz. 

Z Til Jjn 

For each partition jj,, we let 

veWn 

where Wfi is the W-orbit of \x (cf. (2.1)). These polynomials form a C-basis of C[x ±1 ] M/ . Then, there exists 
a unique family of polynomials = P^°(x;ki,k 2 ,k 3 ) G C[a; ±1 ] 14/ (A are partitions such that £(X) < n) 
satisfying two conditions: 

P A HO (x) = m BC (x) + «a m ^ c (x), with « v e C, (P A HO , P m ho ) a ho =0 if A £ M - 

The Laurent polynomials Pa are known as Jacobi polynomials associated with the root system of type BC n 
due to Heckman and Opdam, see e.g. [Di, HS, Mi]. They can be seen as BC-analogues of Jack polynomials. 

For a function / on T", we denote by (f) k l 1 ' k2 ' k3 the mean value of f with respect to the weight function 
A HO (z;fc 1 ,fc 2 ,fc 3 ): 

mfel , fe2 , fe3 = / T „/(^)A HO (^;fc 1 ,fc 2 ,fc 3 )d^ 

From the three parameters k\,k 2 ,k 3 , we define new parameters 

k 1 = k 1 /k 3 , k 2 = (k 2 + l)/k 3 - 1, k 3 = l/k 3 . 

Put 

n 

■* BC (z;x) = ^{{l + xz^l + xzj 1 ). 
Theorem 5.1. The following relation holds 

(5.1) (<f BC (z; Xl )* BC (z; x 2 ) ■ ■ ■ * BC (z; x m )) k ^ MM = ( Xl ■ ■ ■ x m ) n P^ m) {x u ...,x m , ~k u h, h)- 
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In order to prove this, we need the following dual Cauchy identity obtained by Mimachi [Mi] . 

Proposition 5.2 ([Mi]). Let x = (x\, . . . , x n ) and y = (j/i, . . . , y m ) be sequences of indeterminates. Jacobi 
polynomials P^° satisfy the equality 

n m 

II 11^ + x ^ - vj - yJ 1 ) = E (-i)'M HO (*; *i. fe, h)Pf°(y, h,~k 2 , k 3 ), 

i=l j=l AC(ra") 

where A = (n - X' m ,n — X m _ 1 , ...,n- \[). □ 
Proof of Theorem 5.1. We see that 

m n 

* BC (z; Xl )* BC (z;x 2 ) ■ ■ ■ * BC (z; x m ) = { Xl ■ ■ ■ x m ) n J] + ^ + z ^ + 

i=l j=l 

Using Proposition 5.2 we have 

(* BC (z;a ;i )* BC (z; a;2 )---* BC (z;a ;m ))^ fe2 ' fe3 
=(a;i---a; ro ) n £ if °0n, . . . , x m ; ~k u k 2 , h)(P?°(z; h, k 2 , k 3 ))* lMM . 

AC(m") 

By the orthogonality relation for Jacobi polynomials, we have 



{Pf°{z;k u k 2 ,k 3 )) k n ^ k ^ = 



1, if A = (0), 
0, otherwise, 



and we thus obtain the theorem. □ 

Remark 5.1. Using Theorem 2.1 in [Mi], we derive a more general form of equation (5.1) including a 
Macdonald-Koornwindcr polynomial. 

Theorem 5.3. Let 

m—l — 

^ (m;fcl '" 2 ' fc3) = g 2^ + ^m + k 2 + ± +J k 3 y 

The m-th moment of^ BC (z; 1) is given by 

At.bcv T T T \ ?! i- r 1 r(n + fci + 2fc 2 +jfc 3 )r(n + fci+fc2 + 5+jfc3) 

f= r(n + ^ + fc 2 + ^)r(n + ^ + fc 2 + ^) 

Proof. By Theorem 5.1 we have 

(5.2) (* BC (z;ir>^ fe2 ' fe3 = P^l^MMM). 

The special case P BO (l, 1, . . . , 1; fci, k 2 , k 3 ) is known and is given as follows (see e.g. [Di] 1 ): for a partition 
A of length < m, 

Pni,...,l;fc 1 ,fe,fc 3 )=2^l TT (^+/» J " + A!3 ? A<+Aj i Pi " P3 \ +fc3)A< - Aj 



l<i<j<m 
™ (k± _ _ /^J 



(f + fc 2 + P 3 )A 3 (^+P 3 )A 3 
(2p,)2A j 



lr The connection between ours notation and van Diejen's [Di] is given by uq = ki + &2, v\ = £12, v = kz- 
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with pj — (to — j)k 3 + ^ + k 2 . Here (a)„ = T(a + n)/T(a) is the Pochhammer symbol. Substituting (n m ) 
for A, we have 

P^ ) (l m ;k 1 ,k 2 ,k 3 ) 

, . -r-r {k l +2k 2 + {2m-i- ] + l)k 3 ) 2n T r 1 2 2n (fc 1 + 2fc 2 +jfc 3 )„(fcl + fc2 + 5+jfc 3 )r l 



(fci + 2fc 2 + (2m - i - j)fc 3 ) 2 „ 11 (fc 1 + 2/c 2 + 2j/c 3 ) 2 „ 

A simple algebraic manipulation of the first product on the right-hand side of (5.3) yields 
-i-r (fc 1 +2fc 2 + (2TO-i-,7 + l)fc 3 ) 2ra = jJ (fa + 2fc 2 + 2jfc 3 ) 2n 

i<ij< m ( fci + 2k2 + j)h) 2n = f} (fa + 2fc 2 + ifc 3 ) 2n 



and therefore we obtain 



^ ) (l-;h,*..h)-n Jfc(h + h + * + * ) 



(n + h + 2k 2 +jk 3 ) n 



Combining the above result with equation (5.2), we have 

(5 4) (tf BC (z- iD klMM = TT 2 2 "r(n + fci + 2fc 2 + jk 3 )T(n + k 1 + k 2 + \+ jk 3 ) 

^ " f} T(k 1 +h + ^+fk 3 )T(2n + k 1 + 2k 2 +fk 3 ) 

Finally, we apply the formula 

r(2a) = ^r( a )r( a + i) 

to r(2n + k\ + 2k 2 + jk 3 ) in equation (5.4) and we then have the theorem. □ 
Corollary 5.4. It holds that 

^bc (z; l)m ^MM ^ T(jn . ^ . nro (fe 1+ fe 2 ) + i ro(ro _i)^ 

as n — > oo. 

Proof. The claim follows from the previous theorem and the asymptotics of the gamma function (cf (4.3)): 
T(n + a) <~ T(n)n a for a constant a. □ 



6 Random matrix ensembles associated with compact symmetric 
spaces 

Finally, we apply the theorems obtained above to compact symmetric spaces as classified by Cartan. These 
symmetric spaces are labeled A I, BD I, C II, and so on, see e.g. Table 1 in [CM]. Let G/K be such a 
compact symmetric space. Here G is a compact subgroup of GL(N, C) for some positive integer N, and K is 
a closed subgroup of G. Then the space G/K is realized as the subset S of G: S ~ G/K and the probability 
measure dM on S is induced from the quotient space G/K. We consider S as a probability space with the 
measure dM and call the random matrix ensemble associated with G/K. See [Du] for details. 

The random matrix ensembles considered in §6.1, §6.2, and §6.3 are called Dyson's circular /3-ensembles, 
see [Dy, Me]. The identities in these subsections follow immediately from expressions (4.7) and (4.9) (see 
also Example 4.1) . Similarly, identities after §6.4 follows from Theorem 5.1, Theorem 5.3, and Corollary 
5.4. 

Note that the results in §6.1, §6.4, §6.5, and §6.6 are results for compact Lie groups (which are not proper 
symmetric spaces) previously presented in [BG]. 
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6.1 U(n) - type A 

Consider the unitary group U(n) with the normalized Haar measure. This space has a simple root system 
of type A. The corresponding p.d.f. for eigenvalues z\,. . . ,z n of M G U(n) is proportional to A Jack (;z; 1). 
This random matrix ensemble is called the circular unitary ensemble (CUE). 

For complex numbers 771, ... , t]l, Vl+i, ■ ■ ■ , Vl+k, it follows from equation (4.7) that 



K 



Yl dct(7 + ti^M" 1 ) ■ Yl dct(7 + VL+l M) 



i=l i=l 
L K 



U(n) 



n vH*- 1 -,* 1 ) • n * A (^ vl^) ) = n vr • ■ ■ • . ^+*)- 

U=l i=l / nj 2 i= l 



In addition, from equation (4.9) we obtain 



(i det ( 7 + mn u( ) = n / !(n ^ +m) ;, - n 1 t-^t • »" 

\' ^ " /t/(n) 11 (j + m )!( n + j)! l = l(j + m )! 



for any £ e T. 



6.2 U(n)/0(n) - type A I 

Consider the ensemble S{n) associated with the symmetric space U(n)/0(n). The space S(n) is the set 
of all symmetric matrices in U(n). The corresponding p.d.f. for eigenvalues z\,. . . , z n is proportional to 
A Jack (2;; 2) — Ili<i<j<n \ z i ~ z j\- This random matrix ensemble is called the circular orthogonal ensemble 
(COE). We have " 



I L K \ 

{J det(7 + rj^M- 1 ) ■ J] dct(7 + r, L+i M) \ 

\*=1 »=1 / S(n 



S(n) 

= (f[ ^(z" 1 ; O • f[ * A ( 2; = n r,r ■ PffiiVu Vl+k; 1/2). 

\*=1 i=l / n.l i=l 



For £ e T, we obtain 



1 (2j + l)!(n + 2m + 2j + l)! (2j + 1)! 



(| det(7 + £M)| > s(b) = n (2m + 2? + 1)!(n + 2j + 1) , - n (2m + 2, + l)! ' " 



6.3 U(2n)/Sp(2n) - type A II 

Consider the ensemble S(n) associated with the symmetric space U(2n)/Sp(2n). The space S(n) is the set of 
all self-dual matrices in U (2n), i.e., M € S(ri) is a unitary matrix satisfying M = JM T J T with J = ( ^ ) . 
This random matrix ensemble is called the circular symplectic ensemble (CSE). The eigenvalues of M <E S(n) 
arc of the form Z\, Zi, z 2 , z 2l ■ ■ ■ , z n> z n and so the characteristic polynomial is given as det(7 + xM) = 
rij=i(l+ xz i) 2 - The corresponding p.d.f. for Z\, . . . , z n is proportional to A Jack (2;; 1/2) = Ili<i<j<ri \ z i~~ z j\ 4 - 
We have 

(j[ det(7 + ^M- 1 ) 1 ' 2 • 1] dct ( 7 + VL+iM^A 

\»=1 i=l / S ( n ) 

= /n* A (^- 1 ; ?? ri ) .[]*A (2 . ??i+4) \ =n a; p.p ( J^( m ,... ) ^ +Jf; 2). 

\i=l 2—1 / n 4 i=l 
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For £ e T, wc obtain 



(\ detfJ + £M)I 2 "^ - r ( 1 T 1 ) r ( n + TO + 1 T 1 ) 2™ , 

{\det { l+SM)\ ) s(n) - _Q r(m+i ± 1)r(n+i ± 1) (j^.!),,^-!^-.!),! 



6.4 SO(2n + 1) - type B 

Consider the special orthogonal group SO(2n + 1). An element M in SO(2n + 1) is an orthogonal matrix 
in SL(2n + 1,K), with eigenvalues given by Zi, zf 1 , • • • , z„, z" 1 , 1. From Weyl's integral formula, the corre- 
sponding p.d.f. of zi, Z2, ■ ■ ■ , z n is proportional to A HO (z; 1, 0, 1), and therefore it follows from Theorem 5.1 
that 

/ n dct ( j + x iM) ) = U(i+x l ) ■ i n * bc (^; = n ^a+^) ■^° ) (^, ■ ■ ■ , » m ; i, o, i). 

\*=1 / SO(2n+l) *=1 \*=1 / n i=1 

Here . . . , ir m ; 1, 0, 1) is just the irreducible character of SO(2m + 1) associated with the partition 

A. Theorem 5.3, Corollary 5.4, and a simple calculation lead to 

(det(I + M) m ) - 2 m TT T(2n + 2 J + 2 ) 22? " m 2 /2+m/2 

{( ict(i+M) ) so[2n+1) -i ii 2j(2j+1)!!r(2n+i+1) nr=i(2i-i)!! 

in the limit as n — > oo. 

6.5 Sp(2n) - type C 

Consider the symplcctic group Sp(2n), i.e., a matrix M e Sp(2n) belongs to U{2n) and satisfies MJM T = J, 
where J = (Sp o n )' ^ e e ig enva l ues are gi ven by z^z-f 1 ,-"" , z n ,z~ x . The corresponding p.d.f. of 
Z\,Z2, ■ ■ ■ , z n is proportional to A HO (;z; 0, 1, 1) and therefore we have 

I m \ I m \ 0,1,1 m 

(l[det(I + Xi M)\ =(H* BC (z- 1 x l )j ='[[x?-P$° ) (x 1 ,...,x m -,0,l,l). 

\i=l I Sp(2n) \»=1 / n i=1 

Here P^°(x\, . . . , x m ; 0, 1, 1) is just the irreducible character of Sp(2m) associated with the partition A. We 
obtain 

Met(T 4- M) m \ - TT T(2n + 2j + 3) 1 ,» 2 /2+m/2 

(det(l+M) ) Sp{2n) - 11 2i+i.(2j + l)!!r(2n + j + 2) TT£=i (2j - 1)!! 

6.6 50 (2n) - type D 

Consider the special orthogonal group SO{2n). The eigenvalues of a matrix M e SO(2n) are of the form 
Zi, zf 1 , • • • , z„, z^ 1 . The corresponding p.d.f. of zi, z 2 , . . . , z„ is proportional to A HO (z; 0, 0, 1), and therefore 
we have 

I m \ / m \ °' ' 1 m 

/ JJdet^ + ^M)^ = /n* BC (z;^)) =n^-- p (n°)(«i»"-»*r n ;0,0,l). 

\i=l / S0(2n) \*=1 / n l =l 

Here P^°(a;i, . . . , x m ;0, 0, 1) is just the irreducible character of 0(2m) (not SO (2m)) associated with the 
partition A. We have 

(det(I + M)" 1 ) - TT F ( 2n + 2j ) 2? " . n m 2 /2-m/2 

(det(7+M) )50(2n)-ll 2J --i (2i _ 1)!!r(2n + j) U ^-\ 2 j-l)\\ 
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6.7 U(2n + r)/(U(n + r) x U{n)) - type A III 

Let r be a non-negative integer. Consider the random matrix ensemble G(n,r) associated with U(2n + 
r)/(U(n + r) x U{n)). The explicit expression of a matrix in G(n,r) is omitted here, but may be found in 
[Du]. The eigenvalues of a matrix M € G(n, r) C U(2n + r) are of the form 

(6.1) z^zf 1 ,--- ,z n ,2:~ 1 , l,l 1 . 

r 

The corresponding p.d.f. of Z\,z*i, . . . , z n is proportional to A HO (2;; r, |, 1), and therefore we have 

/ rn \ m I m \ r ' 2 ' ^ m 1 

( [] det(J + arjM) ) = JJ(1 + a: i ) r ■ ( J] * BC (*5 ) = IE 1 + x *Y x i ' ■ ■ ■ M r, -, 1). 



G(n,r) 



We obtain 



(det(J + MD G(n r) = 2™ (^ c (z; 1))^' 

7r m /2 |^ T(n + r + j + l) 2 7T m / 2 



• n 7 ™ 2 / 2 + rm 



n.ro^xr+j)! ,i r(n+i±§±i)r(n+^ + i) 2 ™(— dp nr=oV + i) ! 

6.8 0(2n + r)/(0(n + r) x O(n)) - type BD I 

Let r be a non-negative integer. Consider the random matrix ensemble G(n, r) associated with the compact 
symmetric space 0(2n + r)/(0(n + r) x 0(n)). The eigenvalues of a matrix M <G G(n, r) C 0(2n + r) are 
of the form (6.1). The corresponding p.d.f. of Z\, z 2 , ■ ■ ■ , z n is proportional to A HO (2;; |, 0, |), and therefore 
we have 



2 2 m 



n det ( j + x * m ) ) = ri( i + aj <) r • ( n * bc ( z ' ^ ) = ri( i + x iY x i ■ ■ ■ ■ > ^ r ' *> 2 )- 

G(n,r) i=1 \i=l I n *=1 



\i=l 

We obtain 



LI 2 2i+r+i ( 4 j + 2r + 1)!! T(2n + 2j + r + 2) J]™- 1 (4j + 2r + 1) ! ! 



6.9 Sp{2n)/U{n) - type C I 

Consider the random matrix ensemble S(n) associated with the compact symmetric space Sp{2ri) / {Sp{2n)C\ 
SO{2n)) ~ Sp(2n)/U(n). The eigenvalues of a matrix M e 5(n) C Sp(2n) are of the form zi, z-f , • • • jZ^z" 1 . 
The corresponding p.d.f. of zi, z 2 , . . . , z„ is proportional to A HO (;z; 0, \, |), and therefore we have 

mdet(/ + XiM)\ =/n* BC (^;^)) =n^-^°)(^i,---^ m ;0,2,2). 

\i=l / s(„) \i=l / „ »=i 

We obtain 

(det(7 + MD - 17 (» + 2j+3)r(2n + 4j + 5) 1 . n ^ +m 

{ ( + > ~ 11 2 2 J+ 2 (4j + 3)!! T(2n + 2j + 4) 2™ HZiW ~ W 
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6.10 Sp{4n + 2r)/{Sp{2n + 2r) x Sp{2n)) - type C II 

Let r be a non-negative integer. Consider the random matrix ensemble G{n, r) associated with the compact 
symmetric space Sp{An+2r) / {Sp{2n + 2r) x Sp{2n)). The eigenvalues of a matrix M e G{n, r) C Sp(An + 2r) 
are of the form 

Z\,Z\,Z-y , z^ , • • • , z n , z n , z n , z n , 1 , . . . , 1 . 
The corresponding p.d.f. of zi, z 2 , . . . , z n is proportional to A HO {z; 2r, |, 2), and therefore we have 

/ m \ m / m \ ^r, §,2 

/ n det(j + xiMfi* j = + ( n * bc (^; **) ) 

\i=l / G(n,r) *=1 \»=1 / n 

m , , 



We obtain 



4'2' 



( ( } ^"nr^W + ^r + l)!! n^r(n + 1 + 1)^+^ + 1) 

24mr+m 2 +rn 

~n j m =o 1 (4j+4r+l)!! 



m +2mr 



6.11 SO(4n + 2) /C/(2n + 1) - type D Ill-odd 

Consider the random matrix ensemble S{n) associated with the compact symmetric space SO(4n+2) / (SO(An+ 
2) n Sp(4n + 2)) ~ SO(4n + 2)/U{2n + 1). The eigenvalues of a matrix M e S{n) C SO(4n + 2) are of 
the form Zi,zi, z^ 1 , z^ 1 , ■ ■ ■ ,z n , z n , z" 1 , z" 1 , 1, 1. The corresponding p.d.f. of z\, z 2 , ■ ■ ■ , z n is proportional 
to A HO {z; 2, |, 2) and therefore wc have 

I m \ m I m \ ^ ' 2 ' ^ 

/ J] det(J + x.M) 1 / 2 \ = JJ(1 + x,) / J] * BC (^; Xi) j 

\i=l I S (n) *=1 \*=1 / n 

m . j 

= 11(1 + Xi )x? ■ Pffiyixu x m ; 1, --, -). 



4' 2 

i=l 



We obtain 



(detf/ + MD ■ ff r ("+* + i) r ("+*) „ 2? " 2+5m . n ™"+m 

(det(/+M) ) s{n) - nU4j _ 1)]] n r(n +4)r(B+4 + i } nr=i( 4 i-!) !! 

6.12 SO(4n)/U(2n) - type D Ill-even 

Consider the random matrix ensembles 5(n) associated with the compact symmetric space SO {An) / {SO{An)C\ 
Sp{An)) ~ SO{An)/U{2n). The eigenvalues of the matrix M € 5(n) C 50(4n) are of the form 

The corresponding p.d.f. of zi, Z2, . - ■ , z n is proportional to A HO (z; 0, ^, 2) and therefore we have 

/ 771, \ / TYl \ ^ ' 2 ' ^ 

/ det(J + ^M) 1 / 2 \ = / * BC (^; *i) ) = ...,x ra ;0,-H). 

\*=1 / S(n) \t=l / n 
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Hence we obtain 



2 m 2 +m Y( n + 2- + -)T(n + 2 m2+m 

(det(J + M) TO >5(„) - n^(i— T)TT ' g r(n+V)r(n+^) ~ H^W^ ' 



7 Final comments 

We have calculated the average of products of the characteristic moments (IlJLi dct(7 + XjM)). We would 
also like to calculate the average of the quotient 

nr =1 det(/ + x 3 M) 
nUdet(/ + yi M) 

Expressions for these quotients have been obtained for the classical groups (i.e., (fci, &2, ks) = (1, 0, 1), (0, 1, 1), (0, 0, 1) 
in our notation) in [BG], but the derivation of expressions for other cases remains an open problem. 




Acknowledgements. The author would like to thank Professor Masato Wakayama for bringing to the 
author's attention the paper [Mi]. 
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